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Abstract 

Student engagement is critical to student learning, especially in the online environment, where students 
can often feel isolated and disconnected. Therefore, teachers and researchers need to be able to measure 
student engagement. This study provides validation of the Online Student Engagement scale (OSE) by 
correlating student self-reports of engagement (via the OSE) with tracking data of student behaviors from 
an online course management system. It hypothesized that reported student engagement on the OSE 
would be significantly correlated with two types of student behaviors: observational learning behaviors 
(i.e., reading e-mails, reading discussion posts, viewing content lectures and documents) and application 
learning behaviors (posting to forums, writing e-mails, taking quizzes). The OSE was significantly and 
positively correlated with application learning behaviors. Results are discussed along with potential uses 
of the OSE by researchers and online instructors. 


Introduction 

Despite the decrease in higher education enrollment overall, online instruction is still growing. A 
recent report funded by the Online Learning Consortium (Allen & Seaman, 2013) found that 6.7 million 
students (about 32% of all college students) were taking at least one online course. Indeed, in the analysis 
of 2,820 institutions (of a potential 4,527 active degree-granting institutions of higher education in the 
United States), over 69% of chief academic officers felt that online learning is important for the future of 
their institutions. The vast majority of these officers (77%) believe that online learning is as good as or 
better than traditional brick-and-mortar learning. 
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Certainly, research comparing face-to-face and online courses has demonstrated that online 
courses can be as effective as traditional face-to-face courses (Maki & Maki, 2007; Robertson, Grant, & 
Jackson, 2005; Zhao, Lei, Lai, & Tan, 2005). Thus, scholarship has moved beyond comparing online and 
face-to-face classes to exploring ways to enhance teaching and learning in the online environment 
(Durrington, Berryhill, & Swafford, 2006; Gaytan & McEwen, 2007; Levy, 2008; Young, 2006). 
Creating and validating research tools to measure various aspects of the online teaching environment is an 
important part of advancing research about online learning (Roblyer & Wiencke, 2004). Such tools may 
also provide feedback to instructors about individual courses. 

Student engagement is, generally, the extent to which students actively engage by thinking, 
talking, and interacting with the content of a course, the other students in the course, and the instructor. 
Student engagement is a key element in keeping students connected with the course and, thus, with their 
learning (Dennen, Darabi, & Smith, 2007; Kehrwald, 2008; Robinson & Hullinger, 2008; Shea, Li, & 
Pickett, 2006; Swan, Shea, Fredericksen, Pickett, Pelz, & Maher, 2000). Therefore, the ability to 
effectively measure student engagement is necessary for online researchers and instructors. The Online 
Student Engagement scale (OSE) provides that. This paper will discuss the conceptualization of student 
engagement and the previous steps in creating, assessing the reliability of, and validating the OSE. It then 
presents the current study, which further validates the OSE’s ability to measure student engagement. 


Literature Review 

Student engagement has been conceptualized in multiple ways across researchers and disciplines 
(Azvedo, 2015). To explain this abstract concept, the key ideas in the research on online student 
engagement are explored: social construction, the widely applied Community of Inquiry model, and the 
particular importance of student engagement to online learning. 

Social Construction 

Social constructivist theories, such as those created by Vygotsky (1978) and Bandura, Ross, and 
Ross (1961, 1963), posit that we learn through social interaction. Students may perform a set of actions 
by themselves but will perform better when allowed to work collaboratively with others. This difference 
between what students can perform by themselves and what they can perform with others is Vygotsky’s 
“zone of proximal development” (Ally, 2004; Anderson, 2004; Ashcraft, Treadwell, & Kumar, 2008; 
Hrastinski, 2009; Stacey, 2002; Vygotsky, 1978; Woo & Reeves, 2007). In an online discussion, for 
instance, students can help each other by filling in the gaps in each other’s knowledge and/or by 
“demonstrating” particular tasks. The zone is the reason interaction with the instructor and with other 
students is so important to learning. 

Bandura et al. (1961, 1963) illustrated that students can also learn by observing others’ behaviors. 
In the online course, such observational learning may occur when students read arguments posted by 
other students or the instructor. These become “models” for learning. Similar processes could occur for 
shared papers, wikis, and so on. This move toward more active learning and interaction with students is 
particularly important in the online environment, where the challenges of lack of synchronicity (not being 
online at the same time) and lack of placedness (not being in the same geographical location) have to be 
overcome (Anderson, 2004). To overcome these challenges, researchers recommend creating courses that 
encourage three characteristics: social presence, community, and meaningful interaction (Ally, 2004; 
Bigatel, Ragan, Kenan, May, & Redmond, 2012; Dow, 2008; Hill, Song, & West, 2009). 

Briefly, the need for active learning and interaction means that students need to feel as if they are 
dealing with real people (social presence), that they belong in some way with/to this group of learners 
(community), and that they are involved in sharing, negotiating, arguing, discussing, and perspective 
taking (meaningful interaction) (Wang, 2008; Woo & Reeves, 2007). According to social constructivism, 
this type of interaction/engagement is necessary for learning (Ashcraft et al., 2008; Ally, 2004; Bigatel et 
al., 2012; Hrastinski, 2009). Hrastinski (2009) defines online learner participation as “a process of 
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learning by taking part and maintaining relations with others. It is a complex process comprising doing, 
communicating, thinking, feeling and belonging, which occurs both online and offline” (p. 1761). 

Some of the implications that constructivism has for online learning are similar to the guidelines 
it creates for traditional instruction: Learning should be active, allow students to construct their own 
knowledge, make effective use of collaborative and cooperative methods, and be meaningful to students 
(Ally, 2004). In this way, online learning environments promote social presence and community while 
creating meaningful interactions. Under these conditions, significant learning is more likely to occur. The 
Community of Inquiry model (Col) provides a clear framework for applying social constructivist ideas to 
the online learning environment. 

Community of Inquiry Model 

The Community of Inquiry model discusses three “presences” that are necessary for an effective 
community of learners: social presence, teaching presence, and cognitive presence (Akyol & Garrison, 
2011; Akyol & Garrison, 2014; Annand, 2011; Garrison, 2007; Arbaugh, 2008; Garrison & Anderson, 
2003; Garrison & Arbaugh, 2007; Shea et al., 2010; Stodel, Thompson, & MacDonald, 2006). Social 
presence, as discussed earlier, is the ability of learners to share more than “just the facts” and to feel they 
are communicating with real people in cyberspace (Kehrwald, 2008). When social presence occurs, 
students feel they are communicating their emotions and attitudes and interpersonally connecting with 
others (Garrison & Arbaugh, 2007). Online researchers emphasize social presence as a key factor in 
student engagement. Researchers have found social presence to be positively related to students’ learning 
and their sense of being connected within the class (Shea et al., 2006), higher performance on writing 
assignments (Picciano, 2002), and student satisfaction (Dennen et al., 2007). Garrison et al. (2000) found 
that “social presence marks a qualitative difference between a collaborative community of inquiry and a 
simple process of downloading information” (p. 96). Thus, online students need to feel that they are not 
alone in their learning, but connected to a group of learners. We have yet to reach the stage of 
technological sophistication (although we are probably close) when one can feel engaged with nothing 
more than a computer, in the sense of feeling as if one is working with someone else to create knowledge, 
solutions, and so on. Students need to feel that they are working with real people: their peers and 
instructors. Social presence is a necessary but not sufficient component of student engagement (Dow, 
2008). 

Cognitive presence includes the practical inquiry model, which moves students’ 
thinking/discussion from a triggered event that makes them aware of some new idea, concept, or problem 
to exploration of the new information, integration of ideas, and finally to resolution of the problem (Akyol 
& Garrison, 2011). Teaching presence is about course design and organization, discourse facilitation, and 
direct instruction (Akyol & Garrison, 2014; Garrison, 2007). Research has supported the relationship 
between one or more of the presences and perceived learner support and perceived learning (Akyol & 
Garrison, 2014; Arbaugh, 2008); student satisfaction and sense of community (Garrison, 2007); and 
higher order learning outcomes (Akyol & Garrison, 2011). Thus, Col offers a strong model for 
researching about online courses as well as designing effective online learning environments. 

Importance of Student Engagement to Online Learning 

Social construction in general and the Col framework in particular support the need for student 
engagement with content, other students, and the instructor. Swan et al. (2000) and Chickering and 
Ehrmann (1996) agree. Swan and her associates (2000) found that the three factors associated with 
successful course design and students reporting high levels of learning and satisfaction were (1) frequent 
and quality interaction with instructors, (2) a dynamic discussion (interaction with classmates), and (3) a 
transparent interface (easy navigation). Likewise, Chickering and Ehrmann (1996) hold that good practice 
must include student-faculty contact, cooperation among students, and active learning. In general, the 
research about online learning supports the idea that student engagement is crucial to student success 
(Dennen et al., 2007; Kehrwald, 2008; Robinson & Hullinger, 2008; Shea et al., 2006; Swan et al., 2000). 
Creating a learning environment that is cohesive and interactive (Gaytan & McEwen, 2007) ameliorates a 
major issue with online courses: students’ feelings of isolation (Lewis & Abdul-Hamid, 2006; Ortiz- 
Rodriguez, Telg, Irani, Roberts, & Rhoades, 2005; Russo & Campbell, 2004; Song & Singleton, 2004) by 
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providing opportunities for students to create connections with the instructor, students, and course content 
(Young, 2006). 

Defining Student Engagement 

As stated earlier, while there are strong theoretical foundations and a very useful model for 
engagement, student engagement, as a term, is not well defined. Kuh (2003) sees engagement as “the time 
and energy students devote to educationally sound activities” (p. 25). His definition gave rise to the 
National Survey of Student Engagement (NSSE). The NSSE benchmarks five clusters of activities 
indicating student engagement, including level of academic challenge, a supportive campus environment, 
enriching educational experiences, student-faculty interaction, and active and collaborative learning 
(Robinson & Hullinger, 2008). The NSSE perspective on student engagement considers the entire 
collegiate experience, both inside and outside of the classroom. Other measures focus more on student 
engagement within the classroom. One of these is Handelsman, Briggs, Sullivan, and Towler’s (2005) 
measure of traditional classroom student engagement. They found four factors illustrating how students 
devote time and energy in the classroom: skills engagement (keeping up with readings, putting forth 
effort); emotional engagement (making the course interesting, applying it to their own lives); 
participation/interaction engagement (having fun, participating actively in small group discussions); and 
performance engagement (doing well on tests, getting a good grade) (Handelsman et al., 2005, p. 187). 
They see student engagement as containing both affective and behavioral components. 



Figure 1. Affective and behavioral components of engagement. 

Combining social constructivist notions of learning, the Col model, and previous incarnations of 
engagement in the traditional classroom, a description of online student engagement emerges: 
Engagement involves students using time and energy to learn materials and skills, demonstrating that 
learning, interacting in a meaningful way with others in the class (enough so that those people become 
“real”), and becoming at least somewhat emotionally involved with their learning (i.e., getting excited 
about an idea, enjoying the learning and/or interaction). Engagement is composed of individual attitudes, 
thoughts, and behaviors as well as communication with others. Student engagement is about students 
putting time, energy, thought, effort, and, to some extent, feelings into their learning. Therefore, the OSE 
attempts to measure what students do (actively and in their thought processes) as well as how they feel 
about their learning and the connections they are making with the content, the instructor, and other 
students in terms of skills, participation, performance, and emotion. 

Previous Work on the OSE 

Because the previous work of creating and assessing the reliability and validity of the OSE is 
detailed elsewhere (Dixson, 2010), a brief summary is presented here. The OSE was initially created 
using a four-step process: (1) reviewing existing measures of student engagement; (2) conducting a focus 
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group to discuss how those measures would need to be changed for the online environment; (3) creating a 
pilot of that initial instrument; and (4) performing a test of the instrument. 

Existing Measures 

The literature about measuring student engagement indicates three potential measurements: 
Roblyer and Wiencke’s (2004) Rubric for Assessing Interactive Qualities of Distance Courses (RAIQDC), 
Ouimet and Smallwood’s (2005) Classroom Survey of Student Engagement (CLASSE), and Handelsman 
et al.’s (2005) Student Course Engagement Questionnaire (SCEQ). The RAIQDC measures interaction by 
asking students about other students’ behaviors. Given that a good deal of student behavior is not 
accessible in the online environment, particularly students’ affective responses, the RAIQDC was not 
acceptable. The CLASSE asks students to report on their own behaviors both within and outside of a 
particular class, but it was designed to be used with a separate faculty CLASSE so that areas of 
importance to faculty can be matched to areas of importance to students. This allows faculty to see where 
there may be areas of incongruence between what they feel is important and what their students value 
(Ouimet & Smallwood, 2005). While this is a useful tool based on NSSE items, it lacks an explanatory 
foundation (other than NSSE) for inclusion/exclusion of particular items. 

The SCEQ was more theoretically sound. As previously mentioned, Handelsman et al.’s (2005) 
survey does not conceive of engagement as one characteristic or only as behaviors but as based on 
multiple factors: skills engagement (what students “do”); emotional engagement (how connected they feel 
to the course/content, which is especially important in online courses; how applicable they feel it is); 
participation/interaction engagement (interacting with others, enjoying the content/course); and 
performance engagement (students’ desire/goal to succeed in the course). The notion that students get 
emotionally involved and need others to learn fits with social constructivist ideas about learning as well as 
the Col model of social, cognitive, and teacher presence. It also fits with research findings regarding the 
need for meaningful activities and connection in online classes. So the SCEQ held a stronger theoretical 
foundation about engagement and measured not just perceptions of attitudes but also perceptions of 
behaviors. Therefore, Handelsman et al.’s (2005) SCEQ was used to form the basis of the online survey. 
However, since this measurement includes items such as “coming to class everyday” and “raising my 
hand in class,” adjustments to fit the online environment were needed. Table 1 summarizes essential 
shortcomings of each measure. 


Table 1 

Differences Between Existing Measures 

Measure 

Shortcomings 

RAIQDC 

Asks about other students’ behaviors but lacks the ability to tap into affective 


components 

CLASSEstudent 

Designed to be used with CLASSE Facu i tv 

SCEQ 

Needed to be adapted to online environment 


Focus Group 

To make these adjustments, a focus group of five online instructors met to discuss what each of 
the four factors (skills engagement, emotional engagement, participation/interaction engagement, and 
performance engagement) might “look like” in the online environment. They identified a set of 30 
behaviors that might represent Handelsman et al.’s (2005) four factors in an online course (see Dixson, 
2010, for complete description). For instance, items such as “taking good notes in class” were replaced 
with “taking good notes over PowerPoints or video lectures.” Likewise, “participating in small group 
discussions” was revised to “participating in small group discussion forums.” From that process, an initial 
30-item questionnaire was created. 
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Pilot 

The third step in creating the scale consisted of a pilot of the 30 items created by the focus group. 
Participants for this pilot were 31 students in online communication courses at a regional midwestern 
university. The results indicated strong reliability (Cronbach’s alpha = .95) and significant correlation 
with two global items on engagement with the course (r = .73 \P< .01), one global item of social presence 
(getting to know other students), and one global item of teacher presence (getting to know instructor) (r 
= .38; p < .05) (Dixson, 2010). Thus, initial reliability as well as an expert jury (focus group) and 
concurrent validity (correlation with the global items) were supported with the pilot. 

Test 

Once the initial three steps were accomplished, the fourth step was to test the engagement scale 
with a larger and more diverse pool of students. To that end, 186 students across 38 courses from six 
campuses of a large midwestern university (the main campus and five regional campuses) completed the 
OSE. As an incentive for instructor participation, aggregate data was shared with instructors if five or 
more students from a course participated. To alleviate students’ concerns about instructors having access 
to the data, no demographic data was collected. 

Factor analysis yielded four factors: skills, emotion participation, and performance of 19 items 
loading .60 or higher (Dixson, 2010). These 19 items indicated strong reliability (alpha = .91) and 
significant correlation with a course global engagement item (r = .67; p < .001). (See Table 2 for a listing 
of how the items distributed across factors.) As in the pilot, these 19 items were significantly correlated 
with both instructor presence and social (student) presence. An unexpected finding of this study was that 
students who spontaneously reported multiple channels for communicating with other students and the 
instructor (i.e., e-mail, discussion forums, etc.) reported significantly higher levels of engagement. 


Table 2 

Item Distribution Across Factors 

Skills 

Emotion 

Participation 

Performance 

Study regularly 

Put forth effort 

Elave fun in online chats 

Do well on tests 

Stay up on reading 

Find ways to make 

Participate actively in 

Get good grades 


materials relevant 

forums 


Look over class notes 

Apply to my life 

Flelp fellow students 


Be organized 

Find ways to make 

Engage in online 



material interesting 

conversations 


Listen/read carefully 

Really desire to learn 

Post regularly in forum 


Take good notes over 


Get to know other 


readings, PPT, video 


students 


lectures 





Completing this final step meant that the OSE had exhibited face, expert jury, and concurrent 
validity along with reliability, and could serve as a reliable indicator of student engagement in the online 
learning environment. It contained the four factors expected: skills (i.e., staying up on readings, 
listening/reading carefully), emotional (i.e., applying course material to their lives, really desiring to learn 
the material), participation/interaction (i.e., participating actively in small-group discussion forums, 
helping fellow students), and performance (i.e., getting a good grade, doing well on tests/quizzes) 
engagement. One concern remained: There was no external validation of engagement, only students’ 
perceptions. Thus, another step was needed. 
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Next Step 

Since all of the data used to accomplish the previous four steps was self-reported, it was 
conceivable that, while the OSE was consistently and reliably measuring something, it might not be 
student engagement. An external (outside student perceptions) measure was needed to further validate the 
instrument. If the OSE was indeed valid, it should have correlated with actual student behavior within the 
class. While the OSE measures characteristics of student engagement beyond behaviors, behavioral 
engagement is part of overall student engagement and, thus, should be correlated with self-reports of 
student engagement. Another study comparing observable (not self-reported) student behaviors with the 
engagement measure was needed. In the online environment, those behaviors are often tracked by the 
learning management system. 

Current Study 

Because of the need to compare observable behaviors with the engagement measure, the current 
study proposed to correlate the OSE with actual online student behaviors as tracked by course 
management software. The tracked behaviors included reading posts, reading/viewing content (reading 
posted documents or e-mails, viewing links or videos), writing posts or e-mails, and taking quizzes. 
Social constructionists discuss many types of behaviors that occur during the learning process. Students 
observe (or read or listen to) new information, and they then need to process and interpret that 
information before they personalize/apply it and, in short, make it their own and make it fit with their 
view of the world (Ally, 2004). In the online environment, there are definite opportunities for 
observational learning (reading posts and content, watching lectures) and for application/interactional 
learning (posting in response to questions or other posts, taking quizzes, writing papers, etc.). 

So, two distinct types of behaviors were accessible: observation (or taking in content), and 
application (or producing/demonstrating learning). The researcher assumed that in most cases students 
would need to observe (i.e., listen to the lecture so they could apply the ideas to their own lives, read the 
posts so they could consider and respond to the posted materials) before they applied that learning (e.g., 
took the quiz, responded to the discussion forum post, etc.). Thus, students would need to read an initial 
post or question before creating their own post, and they would need to read content and/or view a video 
lecture before taking the quiz. Therefore, online behaviors available from the course management system 
were divided into observation learning behaviors and application learning behaviors. Student engagement 
should be strongly correlated with both types of behaviors, but since applying learning behaviors requires 
the learning behaviors to occur first, the students who do more application behaviors should be more 
actively involved in the course and, thus, more engaged. While the distinctions are somewhat arbitrary 
since learning can and should occur during both types of behaviors, they are meaningful in creating levels 
of engagement within the course in terms of behaviors. 

The researcher is cognizant of the fact that measuring student behaviors via course management 
software leaves out time spent thinking and reading, and that engagement goes well beyond behaviors 
produced with clicks online (indeed, the OSE measures a much broader definition of engagement, as 
presented above). This test is meant only as a method for providing more objective behavioral validation 
of the OSE than might be provided by only having students self-report their behaviors. It is not attempting 
to measure (or claiming to measure) all learning that occurs in an online course. 

Thus, given the Community of Inquiry model and the previous work to establish the validity and 
reliability of this measure, the current study, to further validate the OSE, predicts the following: 

Hp OSE will significantly and positively correlate with observation learning behaviors in an 
online course. 

H 2 : OSE will significantly and positively correlate with application learning behaviors in an 
online course. 
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Method 


Participants 

To test these hypotheses, online communication instructors at a regional campus of a midwestem 
university were asked to forward an e-mail within Blackboard (the learning management system) to their 
students requesting their participation in a study about online learning. This occurred toward the end of 
the spring semester. Students were asked to complete the OSE and to give their permission for their 
instructor to release the tracking information about their online behavior. Thirty-four students (23 females 
and 11 males) from 13 sections of 5 upper level undergraduate communication courses completed the 
Qualtrics survey and gave permission for their tracking information to be used. This represented about a 
10% response rate. The researcher was not instructing any of the participating sections. Five of the 
students were excluded from correlation analysis due to missing data on the OSE. 

Procedures 

Once students had completed the survey and the semester had ended, instructors submitted the 
tracking information for analysis. Tracking information included a semester’s worth of observation 
learning behaviors (number of e-mails, discussions and assignments read, along with the number of web 
pages, content pages, and files viewed) and application learning behaviors (number of e-mails sent, 
discussions posted, assessments finished, and assignments submitted). As discussed earlier, while a 
student has to do something (e.g., log on, click a link) to accomplish observation learning behaviors, there 
is a difference between the student who attends class and one who attends and participates. Observational 
behaviors were differentiated from application learning behaviors. This data analysis examined only the 
quantity of behaviors, not the degree to which those behaviors affected student learning. Students could 
have simply clicked on and opened an e-mail without paying much attention to the content, and they 
could have taken a quiz by guessing at the answers. But for the purposes of seeing if the engagement scale 
is correlated with actual behaviors, the quantity of behaviors, while a crude measure, is still a valid 
estimate of student learning behaviors. Thus, the data analysis distinguished between observation 
behaviors of paying attention to content and application behaviors of actually doing something with the 
content and/or with classmates or the instructor. The observation learning behaviors and application 
learning behaviors were separately summed. 

Students also completed the 19-item OSE (see Appendix A). To do so, they reported on a 5-point Likert 
scale how well each behavior, thought, or feeling was characteristic of them or their behavior. Statements 
included descriptions of thoughts/emotions such as “really desiring to learn the material”; skills like 
“being organized” and “taking good notes over readings, PowerPoints, or video lectures”; participation 
such as “helping fellow students” and “posting in the discussion forum regularly”; and performance items 
like “doing well on the tests/quizzes” and “getting a good grade.” The OSE Cronbach’s alpha for this 
sample was a .86, meaning the instrument demonstrated internal reliability. Results were obtained by 
running two separate Pearson’s correlations: one between the OSE and the number of observation 
learning behaviors and the second between the OSE and the application learning behaviors. 


Results 

Hp The OSE will significantly and positively correlate with observation learning student 
behaviors in an online course. 

This hypothesis was not supported. See Table 3 for correlation statistics. 

H 2 : The OSE will significantly and positively correlate with application learning student 
behaviors in an online course. 

This hypothesis was supported. See Table 3 (next page) for correlation statistics. 
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Table 3 

Statistics for Hypotheses Tests 



Observational behaviors 

Application behaviors 

n = 29 

mean = 1705.91 

mean = 93.58 


sd= 986.78 

sd = 67.91 

OSE 

mean = 4.02 
sd=. 49 

r = .32; p = ns 

r= .48; p < .01 


Discussion 

The finding that the application learning behaviors were significantly correlated with the OSE 
scale strongly supports the validity of the scale in measuring the engagement of students. While the scale 
goes beyond behaviors (measuring emotional and performance engagement as well as skills and 
participation), this study provided evidence that the self-reports of students are correlated with their 
observable (by a course management system) learning behaviors, validating the scale with objective data 
about behaviors. The validity of self-reports of engagement used by the OSE is supported by actual 
behaviors in the online class. Thus, the primary purpose of this study was accomplished. 

While it was expected that both application behaviors and observation learning behaviors would 
be positively correlated with self-reported engagement, only application learning behaviors significantly 
correlated with self-reported engagement. It may be that simply reading posts, e-mails, content, and so on 
is not enough to be “engaged” in the course. So the number of observation learning activities is less 
relevant to engagement unless it is followed up by posting in the discussion forums, answering e-mails, 
and other application learning behaviors, a conclusion that fits with a social constructivist perspective. 
While notions such as observational learning would indicate that students can learn by reading, listening, 
and thinking, Vygotsky’s ideas about the social nature of learning would indicate that only when we 
engage in communication with others about what we are learning do we have the opportunities to “test” 
our skills/knowledge, receive feedback, and engage more deeply with the content. The need for 
interaction to enhance learning fits with what we know about active learning in the classroom. A straight, 
traditional lecture with no room for student interaction will, in most cases, simply not be as engaging as 
some form of active learning (Ghani, 2009; Keyser, 2000; Margurber, 2005). This type of interaction with 
others, if within the zone of proximal development, can help students move that zone in the direction of 
becoming more knowledgeable/skilled within the content area. Therefore, social constructionist 
perspectives are supported with the finding that only application learning behaviors were significantly 
correlated with students’ reports of engagement in online courses. 

The sheer number of observation learning behaviors was, however, somewhat surprising. Thirty- 
four students logged an average of just over 1,700 observation behaviors. This seems an extraordinary 
number of e-mails, posts, web pages, and other files to read, or at least click on, during a 16-week 
semester. The fact that the standard deviation (986.78) was almost one thousand indicates a lot of 
variability from class to class and student to student. It is possible, of course, that many of the behaviors 
reported by the course management software were just “clicks” and skimming of e-mails, posts, files, and 
web pages rather than students actually taking the time to read and consider each one. As previously 
stated, this data does not distinguish the quality of learning, simply the behavior of accessing the 
information. That fact may explain why the observation learning behaviors were not correlated with 
student engagement—they are not necessarily indicative of being engaged with the content or other 
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students/instructors. They are indicative of accessing the potential to be engaged. Observation behaviors 
were, however, significantly correlated with application behaviors (r = .48; p < .001; n = 29). Thus, 
observation behaviors are likely a necessary but not a sufficient factor in engagement. The numbers 
provide a rough indication of the amount of work students are doing before enacting behaviors that 
demonstrate their learning. 

Limitations 

Of course, there are limitations to this study. As always, a larger and more diverse sample would 
lead to stronger and more generalizable conclusions. Given the voluntary nature of the sample, it is 
certainly possible that those who volunteered were more engaged than others. However, since a 
correlation was used, the volunteer nature of the sample should have little effect on the statistical tests 
other than to restrict the variability of results. Accessing students’ learning behaviors in more depth (are 
they taking notes or on Facebook while viewing the video lecture?) would provide a stronger sense of the 
behaviors involved. Future iterations of the scale shoidd pilot more items asking students to consider their 
cognitive activities. At this point, students were asked about finding ways to make the course material 
relevant and applying the material to their lives. Other items measuring cognitive efforts (e.g., “1 spend 
time thinking about the readings, content, etc.”) might better tap into this important component. 

In the previous study (Dixson, 2010) with 186 students from six campuses using multiple 
learning management systems, the mean engagement was 3.41. The current sample was quite a bit higher 
at 4.02, likely due to the smaller size and the use of communication courses that may require more student 
engagement in terms of discussion forums, chats, and so on. Of course, both means may be affected by 
the voluntary nature of the samples. More engaged students are more likely to volunteer for the study. 
Ideally, the next study would be used as part of a regular class and, thus, would require everyone to take 
the study, creating a sample that is not potentially biased by volunteerism. Even given this limitation, the 
data from both studies can begin to provide some benchmarks about student engagement in different 
types of classes that instructors can use to assess the engagement of their own students. 

Potential Uses of the Scale 

The OSE has three primary functions: (1) to aid research into online course design, (2) to provide 
feedback to instructors about the level of engagement of their students given the course design choices 
made, and (3) to provide evidence of teaching effectiveness for merit arguments, teaching awards, and 
promotion and/or tenure cases. 

As a research tool, it has, so far, shown strong reliability and validity. The current study 
strengthens the argument regarding the validity of the OSE scale. If particular activities, designs, or 
teaching methods are said to increase student engagement, the OSE can be used to test those claims. 

If an individual instructor or set of instructors wished to measure the level of engagement of their 
students, it could provide that type of feedback as well. The instrument should be especially useful in 
providing feedback before and after course design modifications to increase student engagement. 

University teachers are increasingly being asked to provide multiple measures of teaching 
effectiveness beyond traditional student evaluations (multiple items with standard Likert scales asking 
how knowledgeable, organized, etc., the instructor was). The OSE is a valid and reliable scale that taps 
into something beyond student satisfaction while still using a student report method. Thus, the OSE may 
be used as an indirect measure of teaching effectiveness. It measures, in more depth than most traditional 
student evaluations, the perceived engagement of students in an online course. Engagement is a necessary 
(but not sufficient) step in student learning given that students must be engaged with the course before 
they can learn. So, while the OSE does not purport to measure learning, it does measure a necessary part 
of the learning environment that teachers work to create. Like student evaluations, the OSE should not be 
used in isolation. Since it measures perceived student engagement, it is vulnerable to all of the factors that 
affect the ability of student evaluations to be used as valid or reliable measures of teaching effectiveness: 
level of the course, type of content, preparedness of students, and so on. However, it provides a measure 
of the environment created by the design choices and responsiveness of the instructor and, thus, is an 
indirect measure of teaching effectiveness. 
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The OSE provides information beyond that available from course management software. Course 
management software offers information about the quantity of behaviors, such as number of e-mails read, 
posts read or written, quizzes taken, and so on. The OSE taps into students’ intellectual efforts, skills, 
performance, and participation as well as the affective/emotional components of learning. 

In conclusion, the OSE scale offers an easy, valid, and reliable way to measure students’ 
engagement in online courses. This information is needed to continue to account for the efficacy of online 
courses, gauge changes in online courses, and move the scholarship of teaching and learning in the online 
learning environment forward. 


References 

Akyol, Z., & Garrison, D. R. (2014). The development of community of inquiry over time in an online 
course: Understanding the progression and integration of social, cognitive and teaching presence. 
Journal of Asynchronous Learning Networks, 72(3-4), 3-22. 

Akyol, Z., & Garrison, D. R. (2011). Understanding cognitive presence in an online and blended 
community of inquiry: Assessing outcomes and processes for deep approaches to learning. British 
Journal of Educational Technology’, 42(2), 233-250. 

Allen, I. E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United 
States. Babson Survey Research Group. 

Ally, M. (2004). Foundations of educational theory for online learning. In T. Anderson & F. Elloumi 
(Eds.), Theory and practice of online learning (2nd ed.). Athabasca, Canada: Athabasca 
University. Retrieved from http://cde.athabascau.ca/online book/index.html 

Anderson, T. (2004). Toward a theory of online learning. In T. Anderson & F. Elloumi (Eds.) Theory and 
practice of online learning (2nd ed.). Athabasca, Canada: Athabasca University. Retrieved from 
http://cde.athabascau.ca/online book/index.html 

Annand, D. (2011). Social presence within the community of inquiry framework. The International 
Review of Research in Open and Distance Learning, 72(5), 40-56. 

Arbaugh, J. B. (2008). Does the Community of Inquiry framework predict outcomes in online MBA 
courses? The International Review of Research in Open and Distributed Learning, 9(2). 

Ashcraft, D., Treadwell, T., & Kumar, V. K. (2008). Collaborative online learning: A constructivist 
example. MERLOT Journal of Online Teaching, 4(1), 109-116. 

Azvedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, 
methodological, and analytical issues. Educational Psychologist, 50( 1), 84-94. 

doi: 10.1080/00461520.2015.1004069 

Bandura, A., Ross, D., & Ross, S. A. (1961). Transmission of aggression through imitation of aggressive 
models. Journal of Abnormal and Social Psychology’, 63(3), 575-582. 

Bandura, A., Ross, D., & Ross, S. A. (1963). Imitation of film-mediated aggressive models. Journal of 
Abnormal and Social Psychology’, 66( 1), 3—11. 


Online Learning - Volume 19 Issue 4 - September 2015 



Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) 


Bigatel, P. M., Ragan, L. C., Kenan, S., May, J., & Redmond, B. F. (2012). The identification of 
competencies for online teaching success. Journal of Asynchronous Learning Networks , 16( 1), 
59-77. 


Chickering, A. W., & Ehrmann, S. C. (1996, October). Implementing the seven principles: Technology as 
a lever. AAHE Bulletin, 3-6. Retrieved from http://www.tltgroup.org/programs/seven.html 

Dennen, V. P., Darabi, A. A., & Smith, L. J. (2007). Instructor-learner interaction in online courses: The 
relative perceived importance of particular instructor actions on performance and satisfaction. 
Distance Education, 25(1), 65-79. 

Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find 
engaging? Journal of Scholarship of Teaching and Learning, 10(2), 1-13. Retrieved from 
http://iosotl.indiana.edu/article/view/1744 

Dow, M. J. (2008). Implications of social presence for online learning: A case study of MLS students. 
Journal of Education for Library and Information Science, 49(4), 231-242. 

Durrington, V. A., Berryhill, A., & Swafford, J. (2006). Strategies for enhancing student interactivity in 
an online environment. College Teaching, 54(1), 190-193. 

Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence 
issues. Journal of Asynchronous Networks, 11(1), 61-72. 

Garrison, D. R., & Anderson, T. (2003). E-learning in the 21st century: A framework for research and 
practice. Retrieved from 

http://portal.ou.nl/documents/89037/89380/Garrison+%26%20Anderson+(2003).pdf 

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: 
Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105. 

Garrison, R., & Arbaugh, J. B. (2007). Researching the Community of Inquiry framework: Review, issues, 
and future directions. Internet and Higher Education, 10, 157-172. 

Gaytan, J., & McEwen, B. C. (2007). Effective online instructional and assessment strategies. The 
American Journal of Distance Education, 21(3), 117-132. 

Ghani, S. A. (2009). Cooperative learning versus the lecture method of instruction in an introductory 
statistics course. Jurnal Sains dan Matematik, 7(1), 59-71. Retrieved from 

http://www.academia.edu/993985/Cooperative Learning Versus The Lecture Method of Instr 
uction in An Introductory Statistics Course 


Handelsman, M. M., Briggs, W. L., Sullivan, N., & Towler, A. (2005). A measure of college student 
course engagement. The Journal of Educational Research, 93(3), 184-191. 

Hill, J. R., Song, L., & West, R. E. (2009). Social learning theory and web-based learning environments: 
A review of research and discussion of implication. American Journal of Distance Education, 
23(2), 88-103. 

Hrastinski, S. (2009). A theory of online learning as online participation. Computers & Education, 52, 
78-82. 


Online Learning - Volume 19 Issue 4 - September 2015 



Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) 


Kehrwald, B. (2008). Understanding social presence in text-based online learning environments. 
Distance Education, 29(1), 89-106. 

Keyser, M. (2000). Active learning versus and cooperative learning: Understanding the differences and 
using both styles effectively. Research Strategies, 77(1), 35-44. Retrieved from 

http://escholarshare.drake.edu/bitstream/handle/2092/251/Keyser%23251 .pdf?sequence=l 

Kuh, G. D. (March/April, 2003). What we are learning about student engagement from NSSE: 
Benchmarks for effective educational practices. Changes, 24-32. 

Levy, Y. (2008). An empirical development of critical value factors (CVF) of online learning activities: 
An application of activity theory and cognitive value theory. Computers & Education, 51, 1664— 
1675. 

Lewis, C. C., & Abdul-Hamid, H. (2006). Implementing effective online teaching practices: Voices of 
exemplary faculty. Innovative Higher Education, 31(2), 83-98. 

Maki, R. H., & Maki, W. S. (2007). Online courses. In F.T. Durso (Ed.), Handbook of applied cognition 
(2nd ed.) (pp. 527-552). West Sussex, England: John Wiley & Sons, Ltd. 

Margurber, D. R. (2005). Comparing student performance using cooperative learning. International 
Review of Economics Education, 4(1), 46-57. 

Ortiz-Rodriguez, M., Telg, R. W., Irani, T., Roberts, T. G., & Rhoades, E. (2005). College students’ 
perceptions of quality in distance education: The importance of communication. Quarterly 
Review of Distance Education, 6, 97-105. 

Ouimet, J. A., & Smallwood, B. (2005). CLASSE - The class level survey of student engagement. 
Assessment Update, 77(6), 13-15. 

Picciano, A. G. (2002). Beyond student perceptions: Issues of interaction, presence, and performance in 
an online course. Journal of Asynchronous Learning Networks, 6(1), 21-40. Retrieved from 
http://sloanconsortium.org/jaln/v6nl/beyond-student-perceptions-issues-interaction-presence- 
and-performance-online-course 

Robertson, J. S., Grant, M. M., & Jackson, L. (2005). Is online instruction perceived as effective as 
campus instruction by graduate students in education? Internet and Higher Education, 8, 73-86. 

Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in 
online learning [Electronic version]. Journal of Education for Business, 84(2), 101-109. 

Roblyer, M. D., & Wiencke, W. R. (2004). Exploring the interaction equation: Validating a rubric to 
assess and encourage interaction in distance courses. Journal of Asynchronous Learning 
Networks, 5(4), 25-37. 

Russo, T. C., & Campbell, S. W. (2004). Perceptions of mediated presence in an asynchronous online 
course: Interplay of communication behaviors and medium. Distance Education, 25, 215-232. 

Shea, P., Li, C. S., & Pickett, A. (2006). A study of teaching presence and student sense of learning 
community in fully online and web-enhanced college courses. The Internet and Higher Education, 
9, 175-190. 


Online Learning - Volume 19 Issue 4 - September 2015 



Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) 


Shea, P., Hayes, S., Vickers, J., Gozza-Cohen, M., Uzuner, S., Mehta, R., Valchova, A., & Rangan, P. 
(2010). A re-examination of the Community of Inquiry framework: Social network and content 
analysis. Internet and Higher Education, 13, 10-21. 

Song, L., & Singleton, E. S. (2004). Improving online learning: Student perceptions of useful and 
challenging characteristics. Internet & Higher Education, 7, 59-70. 

Stacey, E. (2002). Learning links online: Establishing constructivist and collaborative learning 
environments. In S. McNamara & E. Stacey (Eds.), Untangling the Web: Establishing learning 
links. Proceedings from the ASET 2002 Conference. Melbourne, July 7-10. 

Stodel, E. J., Thompson, T. L., & MacDonald, C. J. (2006). Learners’ perspectives on what is missing 
from online learning: Interpretations through the Community of Inquiry framework. The 
International Review of Research in Open and Distributed Learning, 7(3). 

Swan, K., Shea, P., Fredericksen, E., Pickett, A., Pelz, W., & Maher, G. (2000). Building knowledge 
building communities: Consistency, contact and communication in the virtual classroom. Journal 
of Educational Computing Research, 23( 4), 359-383. 

Vygotsky, L. (1978). Interaction between learning and development. In Mind and society (pp. 79-91). 
Cambridge, MA: Harvard University Press. (Reprinted from Readings on the development of 
children, pp. 29-36, by M. Gauvain & M. Cole, Eds., 1997 New York: W. H. Freeman and Co.). 

Wang, Q. (2008). A generic model for guiding the integrations of 1CT into teaching and learning. 
Innovations in Educational and Teaching International, 45(4), 411-19. 

Woo, Y., & Reeves, T. C. (2007). Meaningful interaction in web-based learning: A social constructivist 
interpretation . Issues in Higher Education, 10, 15-25. 

Young, S. (2006). Student views of effective online teaching in higher education. The American Journal 
of Distance Education, 20(2), 65-77. 

Zhao, Y., Lei, J., Lai, B. Y. C., & Tan, H. S. (2005). What makes the difference? A practical analysis of 
research on the effectiveness of distance education. Teachers College Record, 107(8), 1836-1884. 


Online Learning - Volume 19 Issue 4 - September 2015 



Measuring Student Engagement in the Online Course: The Online Student Engagement Scale (OSE) 


Appendix A 

Online Student Engagement Scale (OSE) 

Within that course, how well do the following behaviors, thoughts, and feelings describe you? Please 
answer using the following scale: 

1. not at all characteristic of me 

2. not really characteristic of me 

3. moderately characteristic of me 

4. characteristic of me 

5. very characteristic of me 

1. Making sure to study on a regular basis 

2. Putting forth effort 

3. Staying up on the readings 

4. Looking over class notes between getting online to make sure 1 understand the material 

5. Being organized 

6. Taking good notes over readings, PowerPoints, or video lectures 

7. Listening/reading carefully 

8. Finding ways to make the course material relevant to my life 

9. Applying course material to my life 

10. Finding ways to make the course interesting to me 

11. Really desiring to learn the material 

12. Flaving fun in online chats, discussions or via email with the instructor or other students 

13. Participating actively in small-group discussion forums 

14. Helping fellow students 

15. Getting a good grade 

16. Doing well on the tests/quizzes 

17. Engaging in conversations online (chat, discussions, email) 

18. Posting in the discussion forum regularly 

19. Getting to know other students in the class 
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